Skip to content

fix(agent): follow a parallel Stream's map, filter, reduce and collect lambdas (#1172) - #1209

Merged
PIsberg merged 6 commits into
mainfrom
fix/parallel-stream-lambdas
Nov 8, 2026
Merged

PIsberg merged 6 commits into
mainfrom
fix/parallel-stream-lambdas

Conversation

@PIsberg

@PIsberg PIsberg commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner

Partly addresses #1172 (reference Stream pipelines; primitive streams' intermediate operations stay open).

Why

Only a parallel forEach's action carried the run to the common pool's workers (#1156). A plain counter or a shared ArrayList inside a parallel map lambda was seen only on the caller's own elements, so the commonest parallel-stream mistakes after forEach drew nothing reliable.

A lambda given to Stream.map, filter or peek now waits in a slot. The woven terminal operation (forEach, forEachOrdered, collect, reduce, min, max, the matches, toList, toArray, count, the finds) lends the run to every slot of its pipeline while it runs, and runs its own lambdas like forEach's action. A pipeline ended by an unwoven terminal such as iterator() lends nothing. With no join there to order the caller after the helpers, lending would report reading a result as a race.

A combiner, a reduce operator or a comparator merges what other subtasks built, after a fork and join the agent does not see. Each element now publishes its thread's clock, the caller's included, and those three receive every clock published so far before they run.

How it was verified

  • Tests first in AgentParallelStreamHooksTest: pipeline and terminal lambdas run for the run on helpers, the caller is ordered after a map lambda once collect returns, an iterator() pipeline lends nothing, and every woven form returns the stream's own answer.
  • Two combiner tests failed before the combine rule, reporting a correct three-argument collect and a reduce over objects built on other workers as races. Both now pass, and the class passed three consecutive runs.
  • Corpus, through the real agent, four new idioms: parallelMap_plainCounter and parallelMap_addsToAnArrayList fire PROMPT/HIGH; parallelMap_countsUnderALock and parallelCollect_containerPerSubtask stay silent. 0 false positives, every gate green.
  • Full library suite: 3921 tests, 0 failures. Agent suite green, including SubstitutionHooksArePointsTest.

Checklist

  • A behaviour change ships with a test that fails without the fix
  • A new or changed detector is tested against the buggy code and its correctly synchronized twin, and its reported severity matches what it can actually prove
  • Docs the change makes wrong are updated in this same change
  • Nothing was hand-edited between VIBETAGS-START / VIBETAGS-END markers
  • Public API changes are intentional and carry the right version bump: 21 new public static hooks on AgentConcurrencyUtilHooks, additive, tagged @since 1.14.2. Change the tag if the next release gets another number.
  • Commits an agent authored carry the Co-Authored-By trailer
  • A new dependency was proposed with a reason (none added)
  • CI is green, or the failure is explained below

Anything reviewers should push back on

  • The combine rule receives every clock published so far, not only those of the two subtasks being merged. It can therefore order a combine after a subtask it does not merge. That edge is real (it goes through the map carrying the clocks), but it can hide a race between such a combine and an unrelated subtask.
  • Pipelines are tracked in a synchronized WeakHashMap keyed by the stream, only inside a run.
  • Not followed yet: a primitive stream's map and filter, a stage behind an unwoven one (sorted, distinct, limit, mapToInt), and a Collector's own functions. All of these run unattributed as before and are documented in docs/agent/limitations.md.

Second commit: the clock cap dropped the common pool's threads

Corpus Eval failed on JDK 25 and 26: idiom_parallelCollect_containerPerSubtask, a correct three-argument collect, drew a SharedCollectionDetector finding. By then the corpus JVM had started about 18,000 workers. A happens-before clock keeps 4,096 threads (#840) and dropped the lowest ids first. The common pool's threads have the lowest ids and are still running, so a combine dropped the entry of the pool thread whose container it was merging. The same loss affected the read after a parallel forEach (#1156) in any suite that had started that many threads.

HappensBefore.Stamp.join now drops ended threads first, lowest ids first, then the lowest ids of running ones. Either way it only ever loses an edge.

  • Failing first: AgentParallelStreamHooksTest.aCombinerStaysOrderedPastTheClockCap and HappensBeforeTest.theCapDropsEndedThreadsFirst both failed on JDK 26 before the fix. The first one's full clock also turned three existing ordering tests red in the same JVM.
  • After the fix: library suite 3923 green on JDK 26, static gates green on JDK 21, corpus eval green on JDK 26 in every lane (idiom lane 82/82).
  • Docs: CHANGELOG entry, HappensBefore class and join javadoc. No VibeTags guardrail added: the drop rule is documented where it lives, and no external contract changed.

Third commit: obtrude analysis vs. a record still being written

Test Suite (21) failed in DetectorAnalysisUnderConcurrentWritesTest: CompletableFutureObtrudeDetector.analyze() threw a NullPointerException while another thread was still recording, because the thread name was not set yet and the null went into Map.of. This bug was already there before this PR (the test came with #1150). analyze() now skips a record that is not complete yet. The new test failed without the fix and passes with it, on JDK 21.

🤖 Generated with Claude Code

…t lambdas (#1172)

Only forEach's action carried the run to the common pool's workers
(#1156), so a race inside a parallel map lambda was seen on the caller's
own elements alone.

A lambda given to Stream.map, filter or peek now waits in a slot. The
woven terminal operation (forEach, forEachOrdered, collect, reduce, min,
max, the matches, toList, toArray, count, the finds) lends the run to every
slot of its pipeline while it runs, and its own lambdas run like forEach's
action. A pipeline ended by an unwoven terminal such as iterator() lends
nothing: with no join to order the caller after the helpers, lending would
report the read of a result as a race.

A combiner, a reduce operator or a comparator merges what other subtasks
built, after the JDK's fork and join, which the agent does not see. Each
element now publishes its thread's clock, the caller's included, and those
three receive every clock published so far before they run, so a correct
three-argument collect stays silent.

Not followed yet: a primitive stream's map and filter, a stage behind an
unwoven one (sorted, distinct, limit, mapToInt) and a Collector's own
functions; documented in the limitations.

Tests first: the pipeline and terminal attribution tests, then the two
combiner-ordering tests, which reported a correct collect and reduce as
races until the combine rule. Corpus: four new idioms, the two broken ones
fire PROMPT/HIGH and the two correct ones stay silent, 0 false positives,
every gate green. Library suite 3921 green, agent suite green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.35374% with 23 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...deversity/asynctest/AgentConcurrencyUtilHooks.java 77.89% 9 Missing and 12 partials ⚠️
...deversity/asynctest/diagnostics/HappensBefore.java 93.75% 0 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

PIsberg and others added 4 commits November 7, 2026 14:15
…1172)

Corpus Eval went red on JDK 25 and 26: SharedCollectionDetector reported
the correct three-argument collect (idiom_parallelCollect_containerPerSubtask)
as concurrent ArrayList writes.

The corpus JVM had started about 18,000 workers by then, and a
happens-before clock keeps 4,096 threads (#840), dropping the lowest ids
first. The common pool's threads are started early and keep working, so
their ids are below every ended worker's, and they were the first entries
dropped. A combine on one pool thread received the clock of the pool
thread that filled the container it merges and dropped that entry in the
same join, so the merge looked unordered. The same loss hit the caller's
read after a parallel forEach (#1156) once a suite had met that many
threads.

Past the cap, entries of threads that have ended now go first, lowest ids
first, then the lowest ids of running ones. Any choice still only loses
an edge, never adds one. The model keeps a weak id-to-thread map so the
over-cap join can ask whether a thread is still running.

Verified: the new AgentParallelStreamHooksTest case (a caller that met
4,104 threads, then a correct collect) failed on JDK 26 before the fix,
and its full clock also turned three existing ordering tests red in the
same JVM; HappensBeforeTest.theCapDropsEndedThreadsFirst failed before the
fix. After it: library suite 3923 green on JDK 26, static gates green on
JDK 21, corpus eval on JDK 26 green in every lane (idiom lane 82/82).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Test Suite (21) failed in DetectorAnalysisUnderConcurrentWritesTest:
CompletableFutureObtrudeDetector.analyze() threw a NullPointerException
while another thread was still recording. recordObtrude creates the
future's state before it sets lastObtrudedByThread, and analyze() passed
that null to Map.of, which threw. ifIssue then dropped the run's whole
report.

analyze() now reads the field once and skips a state whose thread is not
set yet. The analysis at the end of the run reports it.

Verified on JDK 21: the new anObtrudeStillBeingRecordedDoesNotBreakAnalysis
failed before the fix and passes after it, with
DetectorAnalysisUnderConcurrentWritesTest green; static gates green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)

Corpus Eval stayed red on JDK 25 and 26: SharedCollectionDetector still
reported the correct three-argument collect
(idiom_parallelCollect_containerPerSubtask) after the clock-cap fix.

From JDK 24 the common pool's workers are innocuous: each time one goes
idle and wakes up again it clears its thread-locals, often partway
through one parallel stream. The thread's clock lives in the CLOCKS
thread-local, so the next access started a fresh clock, whose own count
began again and which had forgotten everything the thread acquired. A
combine on that thread then merged a container it had filled itself and
read the merge as unordered. A CI trace showed it: worker-3's own count
was 380 when it wrote the list, while the clocks it published carried
126 for worker-1, whose clock was at 252. JDK 21 does not clear them,
which is why it never failed.

start() now returns the clock THREADS already holds for the thread, so
the same thread keeps the same clock.

Verified on JDK 25: the new aClearedThreadLocalKeepsTheClock failed
before the fix and passes after it; the full library build with all
gates (3925 + 113 + 2 tests) and corpus-eval (139 tests) are green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ambdas

Takes main's CompletableFutureObtrudeDetector fix (#1219) over this branch's narrower one, and main's copy of its test. CHANGELOG keeps main's #1173 and #1212 entries and adds this branch's three #1172 entries; the branch's obtrude entry is dropped as main's #1212 entry covers it.
PIsberg added a commit that referenced this pull request Nov 8, 2026
…analysis

Brings in #1209's merge of main, which takes main's CompletableFutureObtrudeDetector fix (#1219).
@PIsberg
PIsberg merged commit a0aad7b into main Nov 8, 2026
53 checks passed
@PIsberg
PIsberg deleted the fix/parallel-stream-lambdas branch November 8, 2026 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant